SEO Checklist

The robots.txt file is a simple text file placed on your website’s root directory. It tells search engine crawlers which parts of your website they are allowed or not allowed to crawl and index. This is an essential tool for managing your site's SEO and privacy.

📌 Why Use a Robots.txt File?

• Prevent indexing of sensitive or duplicate pages (e.g., admin panels, login pages).


• Optimize crawl budget by guiding bots to important content.


• Block unwanted bots or web scrapers.

📝 How to Create a Robots.txt File

1. Open any text editor (e.g., Notepad, VS Code).


2. Write your rules using User-agent and Disallow directives.


3. Save the file as robots.txt.


4. Upload it to the root directory of your domain (e.g., https://yourdomain.com/robots.txt).

🔧 Basic Syntax

Here’s a simple example:

# Block all web crawlers from all content

User-agent: *

Disallow: /

# Allow all web crawlers access to everything

User-agent: *

Disallow:

# Block specific folder

User-agent: *

Disallow: /private-directory/

🎯 Target Specific Bots

# Block Googlebot from accessing /test/

User-agent: Googlebot

Disallow: /test/

🔍 Allow Specific Files

User-agent: *

Disallow: /private/

Allow: /private/allowed-file.html

🗺️ Adding Sitemap

Include your sitemap in robots.txt to help crawlers find it easily:

Sitemap: https://yourdomain.com/sitemap.xml

🚫 What Robots.txt Can’t Do

• It can’t stop pages from being indexed if other sites link to them.


• It doesn’t provide security—use proper authentication for that.


• It’s publicly accessible—anyone can see your disallowed paths.

✅ Best Practices

• Always test your robots.txt using Google Search Console.


• Don’t block important content like product pages or blogs accidentally.


• Keep it updated as your site structure changes.

🔗 Learn More

Visit the official guide: Google’s Robots.txt documentation